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ABSTRACT 

This  paper  will  describe  the  progress  in  the  development  of  a  methodology  for 
reliability  assessment  for  SHM  systems.  The  methodology  consists  of  the 
development  and  use  of  appropriate  statistical  metrics  of  reliability  for  SHM 
systems  that  detect,  localize,  and/or  size  damage,  and  a  protocol  designed  for 
utilizing  empirical  data,  models,  simulations,  and  uncertainty  analyses  for 
statistically  characterizing  these  systems.  The  protocol  comprises  several  critical 
steps:  a  procedure  for  analyzing  all  pertinent  characteristics  of  the  SHM  system  to 
identify  the  critical  factors  that  affect  system  performance,  a  multistage  approach  to 
system  validation,  a  modeling  and  experimental  methodology  for  efficiently 
addressing  a  wide  range  of  damage  and  operational  conditions,  and  effective 
methods  for  evaluating  the  appropriate  metrics  of  reliability  depending  on  the  SHM 
system  type  and  function.  A  critical  aspect  of  the  protocol  is  that  by  addressing 
variability  in  the  model  and  minimizing  unexplained  error  in  the  representation,  less 
experimental  data  will  be  required  to  address  the  total  unknowns  in  the  evaluation. 
The  application  of  these  methods  in  the  protocol  will  be  presented  in  detail 
including  several  demonstrations  using  simulation-based  case  studies. 


INTRODUCTION 

To  support  effective  deployment  of  Structural  Health  Monitoring  (SHM) 
systems  that  effectively  complement  Nondestructive  Evaluation  (NDE)  methods  to 
enable  Condition  Based  Maintenance  (CBM)  for  structures,  SHM  systems  must  be 
the  subject  of  verification  and  validation  (V&V)  processes  congruent  with  the  level 
of  reliability  that  the  system  must  achieve  in  detecting,  localizing,  and/or 
quantifying  structural  health  degradation.  Unfortunately,  empirical  assessment  of 
the  perfonnance  of  an  SHM  system  in  its  operational  enviromnent  is  not  trivial  and 
could  easily  be  cost  prohibitive  because  of  the  extensive  testing  required  on 
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representative  structures,  expensive  testing  equipment,  and  custom  made  test 
fixtures  to  establish  statistically  significant  performance  metrics.  The  assessment  of 
nondestructive  damage  detection  systems  typically  requires  a  formal  probability  of 
detection  (POD)  study  as  prescribed  in  MIL  HDBK  1823A  [1],  To  help  mitigate  the 
extensive  amount  of  empirical  data  often  required  according  to  MIL  HDBK  1823 A, 
a  process  for  model-assisted  probability  of  detection  (MAPOD)  evaluation  has 
recently  been  developed  and  incorporated  into  the  latest  revision  of  the  handbook 
[4].  MAPOD  evaluation  leverages  both  computer  models  and  transfer  functions  to 
enable  the  detennination  of  the  sensitivity  of  damage  detection  systems  while 
minimizing  the  need  for  empirical  data. 

Recently,  the  authors  have  developed  a  draft  protocol  for  a  new  model-assisted 
probabilistic  reliability  assessment  (MAPRA)  methodology  for  SHM  [5], 
Leveraging  the  MAPOD  approach,  this  methodology  incorporates  empirical  data, 
models,  and  simulations  for  characterizing  SHM  methods  in  tenns  of  the  proposed 
statistical  metrics  of  reliability  for  damage  detection,  localization,  and  sizing.  This 
model-based  methodology  attempts  to  minimize  the  number  of  samples  that  must 
be  prepared  with  representative  damage  and  testing  to  be  performed  to  obtain  the 
data  required  to  achieve  statistically  meaningful  characterization  results.  This  paper 
describes  progress  on  the  development  of  a  protocol  and  presents  an  initial 
demonstration  study  on  the  MAPRA  methodology.  A  discussion  is  first  presented 
on  the  need  and  general  approach  for  an  SHM  reliability  protocol,  including  insight 
into  mitigating  testing  requirements.  The  design  and  progress  of  an  experimental 
demonstration  study  is  presented  highlighting  protocol  feasibility  and  remaining 
challenges.  A  simulation-based  study  for  an  SHM  system  incorporating  vibration 
methods  is  also  introduced  to  demonstrate  the  protocol  process.  Probability  of 
detection  and  probability  of  correct  characterization  curves  were  generated  for 
different  transducer  locations  providing  key  insight  on  sensor  placement  and 
expected  detection  and  characterization  perfonnance. 


PROTOCOL 
Overview  of  Approach 

To  address  these  challenges  for  SHM  implementation  concerning  extreme 
operation  conditions,  sensor  degradation,  critical  damage  observability,  and 
eliminating  false  calls  due  to  varying  non-damage  conditions,  a  protocol  to  validate 
SHM  damage  detection  capability  is  being  developed  [5],  An  outline  of  the 
MAPRA  protocol  is  given  in  Figure  1.  The  MIL  HDBK  1823 A  for  USAF  NDE 
certification  based  on  POD  is  the  foundation  of  the  protocol.  In  addition,  model- 
assisted  approaches  must  be  applied  to  address  limits  of  experimentation,  facilitate 
proper  uncertainty  analysis  for  damage  detection  cases  and  expand  the  assessment 
to  quality  of  localization/characterization  estimates.  This  protocol  includes  four 
critical  components:  (1)  a  procedure  to  identify  the  critical  factors  impacting  SHM 
system  performance;  (2)  a  multistage  or  hierarchical  approach  to  SHM  system 
validation;  (3)  a  model-assisted  evaluation  process  to  address  the  wide  range  of 
expected  damage  conditions  that  cannot  be  experimentally  tested;  and  (4)  POD, 
probability  of  false  call  (POFC)  and  probability  of  random  missed  call  (POMC) 
evaluations  with  confidence  bounds  estimation  and  uncertainty  analysis  for  damage 


Figure  1.  Outline  of  protocol  for  SHM  validation. 

detection  SHM  systems,  and  evaluation  of  appropriate  probabilistic  metrics  to 
characterize  the  quality  of  damage  localization  and  damage  characterization  for 
SHM  systems  that  include  such  capabilities.  The  multistage  validation  approach  is 
designed  to  incrementally  test  SHM  systems  with  structures  of  increasing 
complexity.  The  multistage  approach  includes  (a)  laboratory  testing  of  relevant 
flaws,  (b)  laboratory  sub-component  testing  including  environmental  and  loading 
conditions,  (c)  a  system  level  life -testing  (full-scale  fatigue  testing  if  feasible),  (d) 
on-structure  demonstration,  and  (e)  final  system  verification.  More  infonnation  on 
the  protocol  can  be  found  in  prior  work  [5]. 

Mitigating  Samples  and  Testing  through  Model-assisted  Evaluation 

A  model-assisted  strategy  for  the  design  and  execution  of  POD  studies  for  NDE 
has  been  developed  and  demonstrated  to  help  mitigate  validation  costs  and  to 
improve  POD  evaluation  quality  by  addressing  a  wider  array  of  inspection 
variables.  By  including  greater  sophistication  in  the  models,  there  should  be  less 
error  present  between  the  model  and  available  experimental  data.  By  addressing 
variability  in  the  model  and  minimizing  unexplained  error  in  the  representation,  less 
experimental  data  will  be  required  to  address  the  unknowns  in  the  evaluation.  The 
following  opportunities  in  the  POD  model  evaluation  and  application  steps  have  the 
potential  to  impact  sample  and  testing  requirements:  (a)  careful  model  factor- 
selection  addressing  system  variation,  (b)  physics-based  model  calibration 
including  uncertainty  bounds  assessment  for  the  specific  inspections  of  interest,  (c) 
controlled  physics-based  model  validation  to  ensure  the  model  is  valid  over  desired 
range  of  application,  (d)  evaluation  of  POD  using  two-level  analysis  to  address 
input  parameter  variability  with  uncertainty  bounds,  (e)  integration  of  experimental 
data  generated  from  a  designed  experiment  using  a  Bayesian  framework  to  revise 
the  prior  distributions  of  inputs  and  achieve  new  posterior  distributions,  and  (f) 
inverse  methods  to  ideally  address  all  uncontrolled  parameter  variations  in  the 
measurement. 

To  address  the  challenge  of  performing  a  quality  evaluation  with  limited  data,  a 
model-based  assessment  must  be  extended,  beyond  a  simple  deterministic 
representation  of  an  NDE  or  SHM  measurement,  to  incorporate  the  variations  of  the 
most  significant  input  factors.  Hoppe  [8]  has  recently  indicated  the  potential  for 
reducing  POD  sample  number  by  better  managing  experimental  variation  using 
more  accurate  statistical  models  and  appropriately  designing  experiments  based  on 
prior  experience  and  data.  Developing  a  validated  stochastic  model  that  includes  all 
significant  sources  of  variation  on  the  measurement  response  is  key  to  sample  and 
experimental  test  reduction.  Simultaneously,  less  costly  studies  could  be  performed 


to  quantify  precisely  the  probability  density  functions  (with  confidence  bounds)  for 
the  key  controlling  factors  a  priori. 

Before  models  can  be  directly  applied  in  POD  evaluations,  two  steps  are  needed 
to  ensure  their  perfonnance.  First,  model  calibration  involves  model  adjustments  to 
variables  in  a  way  that  mimics  the  NDE  technique  procedure.  In  many  cases,  gains 
and  thresholds  are  set  based  on  the  desired  response  to  a  known  calibration 
standard.  Parameters  of  the  model  are  essentially  fit  to  obtain  the  best  match  with  a 
limited  set  of  empirical  data  acquired  according  to  a  calibration  procedure.  Second, 
model  validation  ensures  that  models  are  in  agreement  with  well  controlled  studies 
for  the  appropriate  range  of  conditions  expected  in  the  field.  Note,  uncertainty 
bounds  on  the  measurements  and  numerical  model  error  must  be  tracked  and 
extended  to  any  model-assisted  evaluation. 

Stochastic  models  are  a  means  to  efficiently  represent  stochastic  systems  and 
address  uncertainty  propagation  without  excessive  computational  overhead.  Monte 
Carlo  methods  have  been  previously  applied  to  MAPOD  evaluation  [9].  New 
efficient  methods  such  as  polynomial  chaos  theory  and  probabilistic  collocation 
methods  (PCM)  will  enable  the  greater  application  of  stochastic  models  [10,11]. 
Fundamentally,  this  evaluation  of  uncertainty  bounds  becomes  a  two-level  analysis. 
Parameter  variability  is  associated  with  the  inherent  variability  or  randomness  of  a 
factor.  For  SHM,  environmental  conditions,  material  properties,  and  part  geometry 
will  all  exhibit  some  degree  of  variation  for  each  instance  of  monitoring. 
Alternatively,  uncertainty  in  input  parameters  is  associated  with  imperfect 
knowledge,  often  requiring  the  need  for  more  or  better  quality  data.  In  practice, 
parameter  variability  and  uncertainty  can  be  represented  by  the  random  variables  of 
a  statistical  distribution  and  their  associated  confidence  intervals.  Several  methods 
including  second  order  probability,  maximum  likelihood  estimation,  and 
bootstrapping  can  be  used  to  evaluate  statistical  distributions  with  uncertainty. 
Evaluation  of  the  input  parameter  distributions  with  uncertainty  can  be  achieved 
using  designed  experiments  (ideally),  validated  inverse  methods  (see  below),  or 
possibly  eliciting  expert  opinion. 

In  prior  work  on  model  validation  and  verification,  work  has  explored  the 
propagation  of  variability  and  uncertainty  in  computer  models  and  demonstrated 
agreement  with  limited  experimental  data.  Although  MAPRA  evaluation  is  similar 
in  many  respects,  the  goal  here  is  to  leverage  mixed  simulated  and  experimental- 
based  trends  to  formulate  a  complete  picture  of  SHM  reliability.  Bayesian  methods 
are  necessary  to  incorporate  empirical  data  with  NDE  models  that  include  prior 
information  /  distributions  (with  uncertainty  bounds)  about  the  input  conditions.  A 
generic  example  of  applying  Bayesian  methods  is  to  evaluate  the  posterior 
distribution  of  an  input  parameter  based  on  the  prior  distribution  and  the  likelihood 
function  propagating  empirical  data  through  the  model.  Through  the  application  of 
empirical  data,  the  posterior  distribution  can  be  evaluated,  providing  a  refinement  to 
the  original  prior  distribution.  Numerical  methods  such  as  Markov  Chain  Monte 
Carlo  (MCMC)  methods  and  Bayes  factors  can  be  applied  to  perform  this 
evaluation  [12].  Care  must  be  taken  to  ensure  that  all  assumptions  applied  are  valid. 

Lastly,  there  is  an  additional  benefit  to  be  achieved  by  implementing  SHM 
techniques  that  use  inverse  methods  to  estimate  any  uncontrolled  conditions  in  the 
measurement.  One  case  is  having  an  eddy  current  system  to  determine  liftoff  over 
the  part.  By  doing  so,  the  NDE  technique  can  be  better  controlled.  This  should 


greatly  impact  the  performance  of  the  technique  during  a  validation  study,  as  well 
as  lead  to  less  uncertainty  in  the  POD  result  if  this  estimate  is  used  in  the  model- 
based  evaluation.  NDE  techniques  that  incorporate  inverse  methods  as  process 
controls  should  naturally  be  easier  to  validate. 


SIMULATED  DEMONSTRATION  STUDY 

A  simplified  numerical  model  was  used  to  demonstrate  and  explore  SHM 
reliability  evaluation  for  the  case  of  a  vibration  based  SHM  technique  for  detecting 
large  cracks  in  a  basic  structural  joint.  Of  particular  interest,  this  study  investigates 
robustness  to  varying  temperature  and  fastener  stiffness  over  prescribed 
distributions  and  provides  output  sensor  metrics  as  a  function  of  flaw  size.  A 
simplified  parameterized  FEM  model  was  created  to  represent  an  aircraft  joint 
subject  to  fatigue  cracking  damage  with  varying  temperature  and  fastener  stiffness  / 
contact  conditions  present  (Figure  2).  Parameter  values  and  conditions  for  the  study 
are  given  in  Table  1.  The  simplified  aluminum  plate  structure  was  fixed  to  an 
‘infinitely-stiff  structure  on  the  ends  and  attached  via  steel  fasteners  with  a 
prescribed  stiffness  to  a  second  ‘infinitely-stiff  structure  at  mid  length. 
Distributions  for  the  varying  temperature  and  fastener  conditions  are  given  in 
Figures  3(a)  and  3(b)  respectively.  An  excitation  force  was  applied  near  the  right 
side  for  damage  detection  [Figure  3(c)].  A  through-thickness  crack  of  variable 
length  was  located  at  the  mid  length.  Four  sensor  locations  as  shown  in  Figure  3(c) 
were  considered  in  this  study.  At  each  location,  a  feature  vector  was  evaluated 
containing  the  difference  between  the  frequency  response  function  (FRF)  for  the 
current  condition  and  a  baseline  condition  over  the  frequency  range  of  40  Hz  to  10 
kHz.  The  baseline  condition  was  defined  as  the  ‘no-crack  case’  with  a  median 
temperature  of  55  °F  and  the  fastener  stiffness  set  at  100%.  The  damage  metric  was 
defined  as  the  root  mean  square  (RMS)  of  the  feature  vector. 

To  perfonn  the  stochastic  numerical  model  evaluation,  the  PCM  was  used  to 
efficiently  sample  the  numerical  model  and  propagate  input  parameter  variability.  A 
2nd  order  PCM  model  was  solved  requiring  the  solution  of  6  collocation  points  for 
each  crack  length.  Simulated  results  were  evaluated  for  6  different  crack  lengths, 


Figure  2.  Simplified  geometry  for  representative  SHM  case  study  for  global 

vibration-based  method. 


Table  1.  Parameters  and  conditions  for  simulated  study. 


Parameter 

Properties 

Frequency  range 

• 

40  Hz  to  10  kHz 

Source  Excitation 

• 

single  source  location  [See  Figure  2  for  position] 

Sensor(s) 

• 

four  sensors  in  study  [See  Figure  3(c)  for  positions] 

Crack  (notch)  length 

• 

varying  crack  length  studied  from  0  to  5  inches 

Temperature 

• 

• 

Gaussian  distribution  [See  Figure  3(a)  ] 

Elastic  properties  a  function  of  temperature 

Boundary  conditions 

• 

• 

• 

Fastener  stiffness  varied  over  uniform  range  from  100%  [See  Figure  3(b)  ] 
Fastener  contact  (sides  fixed  to  ground)  remained  constant  in  study 

End  stiffness  conditions  were  fixed  in  study 

thus  requiring  a  total  of  36  numerical  runs  for  the  study.  Figure  5(a)  presents  output 
results  for  all  of  the  collocation  points  and  the  damage  metric  ‘distributions’  as  a 
function  of  transducer  location  and  varying  crack  length.  From  these  distributions,  a 
typical  a-hat  (damage  measure)  versus  a  (crack  length)  relationship  including 
confidence  bounds  can  be  calculated.  The  final  step  is  evaluating  the  POD  curves 
for  each  transducer  as  shown  in  Figure  4(b).  The  results  indicate  that  for  this  case 
the  sensors  positioned  in  close  proximity  to  the  crack  (in  particular,  the  sensor 
located  between  the  crack  and  input  source)  were  most  sensitive  to  the  growing 
flaw.  All  in  all,  this  case  study  provides  a  nice  demonstration  of  the  process  where 
POD  for  an  SHM  technique  in  the  presence  of  varying  conditions  can  be  estimated. 

The  simplified  numerical  model  was  also  used  to  demonstrate  and  explore 
SHM  reliability  for  characterizing  cracks.  The  characterization  ‘model’  tested  here 
is  presented  in  Figure  5(a)  relating  the  damage  metrics  for  select  transducer 
locations  to  flaw  size.  The  characterization  model  is  based  on  a  straightforward  3ld 
order  polynomial  fit  to  the  FEM  numerical  data  for  mean  temperature  and  fastener 
stiffness  conditions.  Probability  of  correct  characterization  was  defined  here  as  the 
ability  to  correctly  classify  the  size  of  a  defect  within  an  acceptable  level  of 
uncertainty  defined  by  8.  Results  for  the  probability  of  correct  characterization  for 
the  case  of  bounds  on  crack  size  accuracy  set  at  8  =  0.49  inches  is  presented  in 
Figure  5(b).  As  with  the  detection  problem,  the  sensor  located  directly  between  the 
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Figure  3.  Input  parameter  distributions  for  (a)  temperature  and  (b)  change  in 
fastener  stiffness  (from  100%  of  steel),  (c)  Map  of  locations  for  4  transducers 
(accelerometers),  input  excitation  and  location  for  crack. 


crack  and  input  source  (transducer  3)  was  found  to  be  most  accurate  for  these 
prescribed  characterization  ‘accuracy’  bounds.  Due  to  the  proximity  of  transducer 
2  to  the  crack  initiation  location,  it  has  more  sensitivity  to  the  smaller  crack  growth 
and  thus  is  exhibits  better  sizing  accuracy  for  smaller  cracks  less  than  2  inches. 


CLOSING  REMARKS 

A  general  methodology  for  validation  of  SHM  systems  has  been  described  and  a 
protocol  for  damage  detection  SHM  methods  was  defined.  The  next  steps  will  be  to 
verify  the  damage  detection  validation  protocol  using  an  SHM  system  of  interest  to 
AFRL,  and  to  extend  the  methodology  to  validate  SHM  systems  that  localize  and 
characterize  damage,  based  on  the  probabilistic  approach  outlined  here. 
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Figure  4.  Results  for  (a)  damage  metric  (including  distributions  generated  by  PCM 
for  varying  temperature  and  fastener  stiffness)  and  (b)  POD  as  a  function  of  crack 
length  for  the  four  transducer  locations. 
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Figure  5.  (a)  Model  relating  damage  metric  to  crack  length  and  (b)  results  for 
probability  of  correct  characterization  for  prescribed  bounds  on  crack  size  accuracy 
of  8  =  0.49  in.  for  transducer  locations  2  and  3. 
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